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Abstract 

Consider the representations of an algebraic group G. In general, polynomial 
invariant functions may fail to separate orbits. The invariant subring may not 
be finitely generated, or the number and complexity of the generators may grow 
rapidly with the size of the representation. We instead study "constructible" 
r^ . functions defined by straight line programs in the polynomial ring, with a new 



"quasi-inverse" that computes the inverse of a function where defined. We write 
straight line programs defining constructible functions that separate the orbits 
of G. The number of these programs and their length have polynomial bounds 
in the parameters of the representation. 
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1. Introduction 

1.1. Background 

When an algebraic group G acts on an affine variety V over a field fc, the 
orbit of a; G y is the set 

G • x = {5 • X I V.g e G}. 

Applications of invariant theory, such as computer vision, dynamical systems, 
and structural chemistry, demand constructive and more efficient techniques to 
distinguish the orbits of a group action. When the group acts rationally, the 
invariant ring 

k[Vf = {fe k[V] I f{g ■ x) = fix) \fg e G} 

contains a finitely generated subalgebra S with the following property: Let 
p,q £ V have disjoint orbit closures, and suppose there exists / e A:[l/]"-' 
such that f(p) ^ /(Q'^- Then there exists h G S such that h{p) 7^ h{q) 



([Derksen fc Kempeii . l2002f ) . We say that the function h (and the algebra S) 
separates the orbit closures of p and q. Note that the functions in S, called 
separating invariants, separate as many orbits as does fc[V^]'^. Since G is an 
algebraic group, G ■ p — G ■ q implies G ■ p = G ■ q, because orbits are open in 
their closures. 

This separating subalgebra S has several weaknesses. For one, existence 
proofs for S may not be constructive for all alge braic groups: K emper's algo- 



rithm to construct S assumes a reductive group (jKempeii 120031 ) . Even in the 
constructive case, although polynomial bounds exist for the d egrees of genera - 
tors for k[V]^ under the action of a linearly reductive group ( Derksenl . 120011 ). 



construction algorithms for separating invariants do not, for general G, provide 
good bounds on the size of a separating subset, the degrees of its elements, or 
the complexity of its computation. Kemper's algorithm, for example, requires 
two Grobner basis calculations, a normalization algorithm, and an inseparable 
closure algorithm. Domokos used polarization to cut down the num ber of vari- 
ables needed in separating invariants of reducible representations (Domokog, 



20071 ). while Kemper provided n ew bouiids, wh en G is finite, on the required 
number of separating invariants (JKempeiJ . 120091 ) . 

As a more serious limitation, the invariant ring fc[T^]'^, and hence any sub- 
algebra, may fail to separate orbit closures. Even when G is reductive, the 



polynomials in A:[y]'^ can separate G ■ p and G ■ q ii and only if G • p n G • g = 0. 
For example, when the multiplicative group G = k* acts on A^ by scaling points, 
one finds fc[a;,2/]'^ = k. 

1.2. Separating Orbits with Constructible Functions 

To overcome the limitations of the invariant ring, we expand the set of regular 
functions on a variety to include a "quasi-inverse" {/} of a regular function /: 

For R = k[V], let R denote the ring of "constructible" functions V ^^ k obtained 
by defining the quasi-inverse on R. For example, ii f, g G -R, then {{/} + (?} G R. 
In fact, one can show that for any ft, G -R, there exists finitely many locally closed 
sets Ei C V and fi regular on Ei such that 



h-J2f^ 



XB, 



where X-E; is the characteristic function of a constructible subset Ei C Spec R. 

For a given group action, we seek to write down a finite set C of invariant, 
constructible functions that separate orbits. That is, if p, q lie in different orbits, 
then some function f E C has f{p) ^ f{q). Even better, we would like the 
evaluation of / at p to be reasonably simple, in the sense of its complexity as a 
straight line program whose inputs are the coordinates of p. 

Over an algebraically closed field fc, fix an embedding of an ?7i-dimcnsional 
algebraic group G ^^ A^. Let R = k[xi, . . . ,Xn], let p : G ^-t- GLn{k) be 
a representation, let r be the maximal dimension of an orbit, and let N = 
max{deg(pij)} be the degree of the representation. 

Theorem 1.1. There is an algorithm, to produce a finite setC C R of invariant, 
constructible functions with the following properties: 

1. The set C separates orbits. 

2. The size of C grows as 0(„27v(^+m+i)(r+i))^ 

3. The f (z C can be written as straight line programs, such that the sum of 
their lengths is 0(n3Af3^(r+i)+r)^ 

Hence the problem of deciding if two points lie in the same orbit can be solved 
with a polynomial number of algebraic operations in the coordinates of the points. 

More explicitly, for p E A" consider the orbit map a-p-. G ^ A" defined by 
Up: g i-^ g ■ p. Note that G • p is defined by the polynomials in the kernel of 
a* : k[xi, . . . ,x„] — > k\G]. These polynomials amount to algebraic relations on 
the images (T*(xi), . . . , cr*(x„) in k[G\. One can find all such relations up to 
some degree d by Gaussian elimination. The coefficients of these relations vary 



with p, but they cannot in general be written as regular functions of p. We may 
nevertheless write them with constructible functions, especially utilizing the 
fact that /{/}(p) = 1 if /(p) 7^ 0. These constructible functions form the set 
C. Essentially, the idempotent constructible functions encode if-then branching 
into the formulas for our relations. 

We proceed in four parts. First, given a matrix X encoding products of 
the <J*{xi) and encoding 1{G) up to some degree d, we produce a matrix of 
constructible functions that gives the entries of the reduced row echelon form of 
X, as functions of p. From these entries follow formulas for the kernel vectors 
of X and hence relations on the a* {xi ) . We next establish a degree bound for 
the relations sufficient to generate the ideal q with V(q) = G ■ p. By considering 
a generating set for q, we provide an algorithm that produces straight line 
programs for the G- invariant functions in the set C. We show that these straight- 
line programs separate orbits and have polynomial length, and we establish 
polynomial bounds for their number in terms of n and the degree N of the 
representation. 

2. Formulas for Reduced Row Echelon Form 

2.1. Straight Line Programs 

Let y be a set, i^ a field, and let R be an F-subalgebra of the F- valued 
functions on V. Let A = (a_m, . . . , a_i) e i?™ be a finite, ordered subset of R. 
Consider a tape of cells with a^ € A in position i. A straight line program F is a 
finite, ordered list of instructions F = (Fq, . . . , F^-i). Each instruction F^ is of 
the form (*; j, fc) or (*; j), where * is an operation and j, k are positive integers 
referring to tape entries in positions i — j and i — k, that is, j and k cells before 
i, respectively. The length i — |F| measures the complexity of the computation. 

To execute F on input A, for i = 0, ...,£— 1 write at in tape position i as 
follows: 



ai^j + ai^k 


ifF, = (+;j, fc) 




ai-j - ai-k 


ifF, = (-;j, fc) 




ai-j ■ ai-k 


ifF, = (x;j,fc) 
if r. = ({•}; j) 


where j,k < 


c 


if Ti = (const; c) for c G i^ 




"i-J 


if F, = (recall; j) 





The "recall" instruction of position j serves to collect relevant computations 
at the end of the tape. Define the order-d output of F by Outd{T,A) = 
{ae-d, ■ . ■ ,ai-i) G -R'^, where £ = |F|. We omit the d where convenient. A 
straight line program hence defines a constructible function R™ -^ R'^. 

Write F^^) o F*^^^ for the composition of two straight line programs, in which 
the input of F^^) is Outd(F(i) , A) for some d depending on F^^) . Then F^^) o F^^) 
has input A, and we execute F^^^ o F^^^ by c o ncate nating the instruction lists. 
For a detailed treatment, see iBiirgisser et al.l (|l997l ). 



2.2. Outline of the Algorithm 

Let A = (aij) he anmxn matrix over a field k. Define the triangular reduced 
row echelon form (tRREF) of A to be tlie n x n matrix Ra = {rij ) whose jth 
row Tj is nonzero if and only if the reduced row echelon form (RREF) of A has 
a pivot in column j. In that case, Tj is the row of the RREF of A containing 
that pivot. For example, 



RREF(A) = 




corresponds to tRREF (^) = 




This new form simplifies the identification of pivots: the (usual) RREF of A 
has a pivot in column j if and only if rjj = 1 in the tRREF. 

Proposition 2.1. Let (aij) be an m x n matrix with entries in any field k. 
Then there exists a straight line program F*^ of length 0(mn? + n^) such that 
(9Mf„2(r*^, {aij)) are the entries of the triangular RREF of {aij). The program 
gives constructive functions for these entries in terms of the Oij . 

The proposition does not require k to be algebraically closed, but we will 
need this condition for the later geometric reasoning about orbits. Note also 
that while the classical Gausssian elimination algorithm requires branching, the 
straight line program F*^ simulates branching in the computation of the quasi- 
inverse. The psuedo-code below proves the proposition in general terms; the 
subsections that follow provide specific constructions. 

Algorithm 2.1. Let A ~ {oij) be an m x n matrix. 

1. For i = 2, ... TO, if an = 0, exchange the first row of A with the ith row. 
After these steps, either an ^ 0, or an = for all i. 

2. Multiply an by {an}, and multiply the rest of the first row by 

(1 — an{aii} + {an}). This is equivalent to dividing the first row by an 
if an 7^0. 

3. For i — 2,..., TO,, subtract an ■ (an, . . . , ai„) from row i. As a result, 
an — for all i > 2. 

4. Let A' = {aij)j>2 and A" = {aij)i_j>2i as below: 



A 



( * 

A' 


\ 




/ * ••• * \ 




Vo 


/ 




; A" 



Let Aq be the m x (n — 1) matrix formed by appending a row of zeros to 
the bottom of A"; then A' and Aq have the same dimensions. 
5. Define B = {1 - an) ■ A' + an ■ A^'. 




6. Recursively compute the tRREF of B; call it Rb, an (n — 1) x (n — 1) 
matrix. 

7. Let Ra be the n x n matrix below: 



Ra 



8. Let Tk be the kth row of Ra = [fij). For k — 2, . . . ,ri, subtract aik ■ fk 
from the first row of Ra- This reduction produces the triangular RREF 
of ^. 

The following formulas specify straight line programs for the entries of the 
triangular RREF matrix Ra, and hence define F*^. 

2.3. Formulas for Gaussian Elimination 

Recall that the first step of the algorithm exchanges the first row of (a^) 
with the iih row if an = 0, for i ~ 2, . . . ,m. Hence for an m x n input matrix 
X, this step requires tti — 1 programs Ei such that Y = Outmn(Si, X) flips the 
first and ith rows if necessary. The following formulas describe the entries of 

Y={V^J)■. 



yii = xu + (1 -a;ii{a;ii})a;ii 

yij == xij + (1 - xiilxii}) • {xij - xij) for aU j > 1 

yn = Xii ■ a:;ii{a;ii} 

yij = xij + a:;ii{xii} • (x^ - xij) for all j > 1 

ykj — Xkj for all fc 7^ 1, i, and for all j. 

For example, the straight line program for yn in Ei takes inputs xn in position 
-2 and xn in position -1, and then performs the following steps: 



(0) 


{•};2) 


(1) 


x;3, 1) 


(2) 


const; 1 


(3) 


-;i,2) 


(4) 


x;l,5) 


(5) 


+;7,i) 



The formulas for the other yij have similarly obvious representations as straight 
line programs. If we concatenate these programs within Ei, so that all the 
entries of Y appear in various (known!) positions on the tape, then we can 
save the recall steps for the end, and we need only compute {xn}, xii{xii}, 
(1 — xii{xii}), and (x^ — xij) once. With these efficiencies, the program Ei 
introduces 1 quasi-inverse, 1 call to fc, 3n additions, and 2n multiplications. 
Thus the concatenation of E2, . . . , E„i~-i requires 2n{m — 1) multiplications. 



3n{m — 1) additions, n — 1 calls to k, n — 1 quasi-inverses, and mn recalls to 
collect the entries of Y in the last mn cells of the tape. Call this concatenation 
r^; we will use it later to collect nonzero rows of a matrix. 

Step (2) of the algorithm requires 1 quasi-inverse, 1 subtraction, 1 addition, 
n multiplications, and n recalls. 

These next formulas perform step (3), on an m x n input matrix (xij): 

yn — for alH > 1 

Vzj = Xy - xij ■ Xii ■ xiijxii} for all i,j > 1. 

These programs require (m — 1) (n — 1) additions, (n — 1) (m — 1) multiplications, 
and mn recalls. Step (5) next requires 1 subtraction, m(n— 1) additions, 2m(n^ 
1) multiplications, and m{n — 1) recalls. 

To perform the reductions in step (8), consider the following formula for rij, 
where j > 2: 

ri'j ■■= (1 - Tjj) ■ {rij + ( - r22 • ?'i2?'2,j 

This formula sets rij = if there is a pivot in column j, that is, if tjj = 1. 
Otherwise, the formula subtracts from rij the effects of clearing columns < j. 
The reduction of rij requires 1 call to fc, j additions/subtractions, 2{j — 2) + 1 
multiplications (since j > 2), and n^ recalls, so reducing the first row has total 
complexity 0{n^). 

The above formulas specify a straight line program F*^ such that Out„2 (F*^, A) 
are the entries of the tRREF of A. Counting the necessary operations yields 
asymptotic total complexity estimates for the programs. The recursion on an 
m X t matrix has total complexity 0{mt + t^). Summing t from 1 to n yields 
total complexity 0{mn^ + n^). 

2.4-- Collecting Nonzero Rows 

Lastly, the main algorithm that computes orbit closures requires a program 
S that, given an indicator vector v of Os and Is, collects the rows z of a matrix 
such that the zth entry of w is 1. For example, the diagonal of Ra indicates the 
nonzero rows of Ra- Given Ra and its diagonal as input, the program E would 
output an n X n matrix whose first rank(yl) rows include the traditional RREF 
of A. We will never need to compute the traditional RREF in practice, because 
the main algorithm runs more efficiently using Ra- 

Recall the algorithm F-^ that exchanges the first row of a matrix X with 
subsequent rows until the output has yn ^ 0, if possible. Define E as follows: 
for an TO X n input matrix X and an indicator to- vector f , form a new matrix 
X' by adjoining w as a column to the left side of X: 



X' 



V2 a;2i • • • a;2n 



After applying F^ to X' , the first row of X' with w^ 7^ becomes the first row 
of the output Y = {nij). Record ri :— (yi2, ■ • ■ , J/i,ra+i) and apply T^ to the 
last TO — 1 rows of this Y . Let S denote this series of to recursions of T^ . Since 
r^ applied to an s x (n + 1) matrix has total complexity 0{sn), the procedure 
E has complexity 0{m?n). Concatenating E with the straight line program for 
the tRREF yields the following: 

Corollary 2.1. Let (aij) be an m x n matrix with entries in any field k. Then 
there exists a straight line program of length 0{mn^-\-m'^n+rr') for the (classical) 
RREF (jij ) of (oij ) . The program gives constructible functions for rij in terms 
of the ttij . 

2.5. Gom,puting Kernels of Linear Maps 

To compute the kernel up to degree d of a fc-algebra homomorphism, one 
can write the homomorphism on elements of degree < d as a matrix in RREF. 
Finding the kernel of a matrix R in RREF is equivalent to solving the system 
of equations R ■ (xi, . . . , x„)"^ = 0: for every pivot r^, write an equation 



'^i^j+lXj+i - rij+2Xj+2 



n. 



Set each free variable equal to 1 in turn, set the other free variables to 0, and 
read off the vector of values in the pivot variables. These vectors give a basis 
for the kernel of R, hence of the original map. The basis is canonical because 
the RREF is canonical. 

To compute the kernel of an ?Ti x n matrix A, we use the n x n matrix 
Ra containing the rows of the RREF of A: recall there is a pivot in the jth 
column of the RREF if and only if the row containing that pivot is jth row of 
Ra — (rij), if and only if rjj ~ 1. Otherwise, rjj = 0. 

Lemma 2.1. Let Ra be the n x n tRREF of a matrix A. Then there exists 
a straight line program, T^ of length 0{n^) such that Outn^{T^ ,Ra) gives the 
kernel of A. 

Proof. Claim that the kernel of A, is given by the following vectors (pi, . . . ,4)m 
in terms of Ra = {rij ) : 

jth place 

4>3-^0'-rn)-{-rij,-r2j,..., 1 ,...,-r„j). 

Indeed, recall that the kernel of a RREF matrix has one basis vector for each 
non-pivot column. Namely, (pj = if and only if column j of the RREF has 



a pivot. Otherwise, (j)j ^ 0, as follows: Put the free variable Xj :— 1. Now, 
rkj = unless there is a pivot in column k of the RREF. Set each pivot variable 
Xkk equal to the negation of the jth entry of the row containing that pivot. 

Of course, Vij — whenever i > j, but such simplifications complicate the 
formulas without improving the asymptotic complexity. As written, each (jjj 
requres 2 calls to fc, 1 addition, n scalar multiplications, and n other multipli- 
cations. Upon adding recall instructions, computing the kernel has complexity 
0(n2). 

3. Degree Bounds for Orbit Closures 

We relate the degree of a variety to the degrees of polynomials that can 
define that variety By bounding the degree of an orbit closure G ■ p, we can 
bound the degree of the defining polynomials. 

Lemma 3.1. Let V = V(/i,...,/r) have codimension m in A". Then there 
exist m generic linear combinations gi — "^aij/j such that 

W:^W{gi,...,gm)^V 

and W has codimension m. 

Proof. Induct on the number r of given defining equations for V. The case 
r = 1, implying m — 1, is clear. Assume the lemma holds for a variety defined by 
r — 1 equations, and consider V = V(/i, . . . , fr-i)- If V still has codimension 
TO, then the result follows by the induction hypothesis. Otherwise, V' has 
codimension m — 1. By the induction hypothesis, there exist m ~ 1 generic 
linear combinations gi of /i, . . . , /^-i such that W = V(gi, . . . , gm-i) ^ V 
and W has codimension to — 1. 

Since W is defined by to, — 1 equations, every component Zk of Y' has 
codimension to.— 1. It follows that on each Zk, one of /i, . . . , /^ is not identically 
zero. So for each Zk, and for every point p e Z^, we may consider the proper 
hyperplane Hk^p C A'' defined by the vanishing of 

Xl.fl{p) + ^ihip) + • ■ • + Xrfr{p) e fc[xi, . . . , Xr]. 

Let Hk = r\p£ZkHk,p- Then UkHk is a closed union of finitely many subspaces 
of A''. Thus for any choice of {ai, . . . ,ar) in the dense set A'' — UkHk, the 
polynomial g„i — "^CLifi is not identically zero on any Zk- Therefore Y = 
V(gi, . . . , gm-i, gm) contains V and has codimension to,. 

Let V C A" be an equidimensional affine variety of codimension m. Define 
the degree of V to be 

deg{V)=#HnV, 

where H is a generic linear subspace of dimension m. 



Proposition 3.1. Let V C A" be a Zariski closed subset of degree d. Then there 
exists an ideal q, generated by polynomials of degree < d, such that y^ ~ I(^)- 
In particular, V(q) = V . 

Proof. It suffices to find, for every point p ^V , a. polynomial / of degree < d 
such that / vanishes on V but not at p. If F is a hypersurface, then V — V(/) 
with deg(/) = deg(T^), and we are done. Otherwise, assume V has codimension 
greater than 1. Without loss of generality, further assume that p is the origin. 

To find a polynomial vanishing on V but not at the origin, we project V 
until an image has codimension 1. Define tt: A" — > P""! by tt: (xi, . . . ,Xn) i— >■ 
[xi : . . ■ : Xn] - Since dim7r(V^) < di mV^ < n — I, t here exists a poi nt [L] € 
P""i - tt(V). Let C{V) = tt-^{tt{V)), the cone over MV). Then L = tt-^{[L\) 
\iasLr\C{V) = {0}. 

Assume without loss of generality that L is the a;„-axis, and consider the 
projection (/>: A" ^ A"^^ along L, defined by 0: (xi, . . . ,x„) i->- (xi, . . . ,a;„_i). 
Because C{V) is a cone, the restriction of (j) to C{y) is a finite map onto A""^. 
In particular, 4){V) is closed in A"~^. Since L is disjoint from V^ 4){Q) = 
remains outside the closed set (jiiV). 

Continue projecting until </> : A" — )• A"~™+^ gives ({){¥) with codimension 1 
(and dimension dimF after each projection). Now, deg(0(F)) < d. Thus there 
exists a polynomial / of degree < d such that / vanishes on (j){V) but /(O) ^ 0. 
Hence / o (^{V) = but / o 0(0) 7^ 0. As </) is defined by linear polynomials, the 
polynomial / o has degree < d, and the result follows. 

Now consider an algebraic group G acting on affine n-space. When we can 
bound the degree of an orbit closure G • x, then we can produce a degree bound 
for polynomials fi such that G ■ x = V(/i, . . . , fr). For an overview of bounds 
for the degr e es of orbits and the (polynomial) degrees of generating invariants, 
see lDerksenl(|200l[ ). 



Proposition 3.2. Let G be an algebraic group of dimension m, embedded 
A^ with ideal 1(G) — {hi, . . . , hs). Set M — max{deg(/ii)}. 
Suppose G acts on A" with representation 

p: G ^ GLn defined by p: g ^ {pij{g)), 



and set N = max{deg(/9ij)}. If G ■ x is an orbit closure with dimension r, then 

deg(G~^) < 7VM^~". 



Proof. Let d — deg(G • x). For a generic (n ~ r)-dimensional linear subspace 
H C A", by definition d = fl^{G ■ xDH). Let a: g 1— ?> g-x be the orbit map. Then 
the degrees of the polynomials defining a are bounded by N. Hence a^^(II) = 
V(ui; . . . , Ur) C G has deg(Mi) < N and has > d irreducible components. 

By the first lemma above, there exist generic linear combinations /-,- of the 
generators of 1(G) such that Y{fi, . . . , fe-m) is a complete intersection and 
contains G. Thus 

a-'iH) C V(ui, ...,ur,fi,..., fe-m) C A^ 
10 



Consider the vanishing of the homogenized polynomials 

By a generalization of Bezout's theorem (see iFultonI ( 19841 ) . section 12.3.1), the 



number of irreducible components of this variety is (generously) bounded by 
ndeg(V(«,)) • ndcg(V(7,)) = ndcg(wO ■ ndeg(7,) < N'-M'-"^. 



This number then also bounds d. 



Corollary 3.1. With the hypotheses of the previous proposition, there exist 
polynomials fi, . . . , ft such that G ■ x = V(/i, . . . , ft) and 



deg(/,) < deg(G • x) < N^M'-^. 

4. Separating Orbits 

Let p : G ^^ GLn act on A" as in Section 3. For p S A", there exists an 
ideal q such that V(q) = G ■ p and q is generated in degree < N'^M^~'^. We will 
establish straight line programs for the orbit-separating set C by considering a 
generating set for q. We prove that these programs define invariant functions 
separating the orbits of G. The length of these programs will be polynomial in 
the dimension n and the degree N of the representation. 

4.I. The Orbit Separating Algorithm 

Input the embedding of G ^-> A^ and the orbit map Cp : g i~> 5 • p as above, 
which varies with p. Let fc[a;i, . . . ,x„] be the coordinate ring of A". Then 
kercr* — I(G • p), but to define G • p it suffices to compute a fc-basis for kera* 
up to degree N'^M^~"^. The elements of this fc-basis generate q as an ideal. 

For each i = 1, . . . ,N^M^~"^, the following algorithm computes a canoni- 
cal fc-basis for kera* in degree < i, but for each polynomial in the basis the 
algorithm only outputs constructible functions (of p) that give the monomial 
coefficients appearing in that basis, whatever the monomials may be. Hence 
the algorithm forgets the generating set of the ideal q. This forgetting allows 
the algorithm to have polynomial length as a straight line program, since the 
number of possibly monomials grows exponentially with n. 

In the most precise sense, given a point p £ A", the following algorithm 
concatenates straight line programs to output a G-invariant vector C over k. In 
fact, each entry of C is a straight line program in terms of the coordinates of p. 
Thus the algorithm prescribes a vector C of G-invariant constructible functions 
that separate orbits: points in distinct orbits produce distinct vectors. The 
proofs for the G-invariance and orbit separation will follow. 

Choose a monomial order for the monomials spanning k[zi, . . . , zg]. As a 
preliminary calculation, compute a Grobner basis and a fc-basis for 1(G) up to 
degree A^''+^M'^^™. Let B{d) denote the set of elements of the fc-basis up to 



11 



degree d. Also, for a vector w, let TTt{w) denote the vector of the the first t 
entries of w. 

Lastly, since all computations occur in k[G], we must predict the dimension 
offc[G]<d. 

Lemma 4.1. Let m — dimG. There exists a function H(d), computable from 
a Grobner basis for 1(G), such that H{d) ~ dimfefc[G]<d for all d > 0, and 
H{d) < Did""). 

Proof. Suppose R = k[G] is generated as a /e-algebra by /i, . . . , /^ of degree 
1. Define S = k[fit, . . . ,frt,t] C R[t], and claim Sd = R<d ■ f^, where 5* is 
graded by i-degree. The inclusion D is clear, and if /i e 5^ is a homogeneous 
polynomial in t, then the coefficients of t''' can have i?-degree no greater than 
d (less, for example, in the term fit ■ t'^~^). Let H{d) be the dth coefficient of 
the Hilbert scries of S, which we may compute from a Grobner basis for 1(G). 
Then H(d) = dirufe R<d- Since S has dimension bounded by tti + 1, the Hilbert 
polynomial for S has degree bounded by m. Thus H{d) < 0{d"^). 

Algorithm 4.1. 

1. For j = 1, . . . , n, let Vj be the vector of coefficients of (yl{xj) with respect 
to the (ordered) monomial basis of fc[zi . . . , zg]. 

2. Vi := {vi,...,Vn). 

3. i:=l,Co = 0. 

4. Put the vectors of V^ — (wi, . . . , wj.. ), in order, in the first ki columns of a 
matrix Xf, fill subsequent columns with B(iN). 

5. Compute Out(r*^,Xi), the tRREF X^. 

6. Compute /3 := Out(r^, Out(r*«,X,)), a basis for kerX,. 

7. Let Ci := C,_i U {nk,{v) \vel3}. 

8. IF iVM^-" = i, THEN output C = d, and STOP. 

9. Let Y be the matrix whose rows are the vectors in Vi. Let D be the first 
ki entries on the diagonal of the tRREF Xi . 

10. Compute Y' := Out(E, {Y, D}), the rows of Y indicated by D. 

11. Let Li be the first H{i) rows of Y'. 

12. IF ki = #(rows of Y) < H{i), THEN pad Li with zeros so that Li has 
precisely H{i) vectors. 

13. V,+i := L, U {W;{xi), ..., a;{xn)} ■ {vj E L, \ j > H{i - 1).}) . 

14. i ■.= i + l. 

15. GOTO (4). 

The final steps of each iteration require some remarks. For step (10), recall 
that the nonzero entries of the diagonal D of the tRREF of Xi indicate which 
columns of Xi are linearly independent. These are the image vectors the al- 
gorithm should preserve for the next iteration, so that it can proceed with a 
polynomial number of multiplications. In step (13), we multiply the a*(xi) only 
by these newfound vectors. 
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Step (12) can be accomplished in the context of straight hne programs be- 
cause we can predict the iteration i at which ki > H(i) first occurs, independent 
of the choice of p. At step (13) we multiply Li by all cr*{xi) because, in principle, 
all (Jp{xi) could be linearly independent modulo 1(G). As i increases, the vec- 
tors in each Vi describe the images of larger monomials x^ , I a. multi- index, in 
k[xi, . . . , Xn]- The algorithm terminates when we have considered a fc-basis for 
the polynomials of degree up to N^M^~"^ that vanish on G ■ p. By the previous 
section, the elements of that fc-basis generate an ideal whose radical is I{G ■ p). 

Proposition 4.1. The constructible functions defined by the set C 

1. are constant on the orbit o/p G A", and hence invariant under the usual 
action g ■ f(x) = f{g-^ ■ x) for g eG, 

2. separate orbits. 

Proof. To show that the functions defined by the straight line programs in C 
are invariant, choose p G A" and q Cz G ■ p. Let Xi(p) be the matrix produced 
in step (4) of the algorithm in the ith iteration. Let X^ (p) be the first |Vi| = /ci 
columns of Xi{p), that is, those containing the vectors in Vi{p). Now, XY (p) 
and XY (q) have the same kernel, because (a) as maps k[xi, . . . , x„]i ^- k[G]<N 
they have the same basis xi, . . . , x„ for their domain, and because (b) the kernel 
of each matrix must span 1(0 • p) i . Thus XY (g 'P) = A- XY (p) for some matrix 
A. In particular, XY{p) and XY (q) have linearly independent columns in the 
same places, and hence have the same RREF. 

So letting Ci{x) denote the kernel vectors obtained on input x in the rth 
iteration, we have Ci{p) = Ci{g ■ p). As well, let Li{p) denote the set (pro- 
duced in step (11) of the algorithm) containing the linearly independent columns 
of XY{p). Then we have Li{p) = {a*{xj^), . . . ,a*{xj^)} and Li{g ■ p) = 
{<^*g-p{^ji)^ • • • : <^*g-p{^or^)} for ttic Same indices ji, . . . , js- 

Proceed by induction on i: we may assume XY {p) and XY {q) have the same 
RREF and hence Ci{p) ~ Ci{q). We may also assume the columns of XY {p) 
and XY {q) represent the images of the same set of monomials {x^^, . . . ,x^=}, 
for multi-indicics Ij. Then the lists ViJ^i{p) and Vi+i{q) also represent the 
images of the same monomials under a* and a* respectively. Claim again 
that XY^i{p) and XY^i{q) have the same RREF. By the induction hypothesis, 
the two matrices have the same basis for their domain, and the kernel of each 
must span I{G ■ p)i+i. These facts prove the claim, as in the base case. Thus 
Ci+i(p) = Ci+i(g), and the functions in C are invariant. 

To show the functions in C separate orbits, choose p,q G A" such that the 
functions in C take the same values at both points. In particular, Ci(j}) = Ci{q), 
so Xi{p) and Xi{q) have the same canonical kernel. As above, it follows that 
Xi{p) and Xi{q) have the same RREF. Two facts emerge. Crucially, the kernels 
of a* and a* have the same canonical /c-basis for their subspaces of degree- 1 
elements, because the matrices XY{p) and XY{q) assume the same basis for 
the domain space fc[xi, . . . , x„]i, namely, xi, . . . , x„. We wish to prove this for 
all degrees i. 
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What is more, Li{p) = {cr*(xjj, . . . ,cr*(xjj} andii(g) = {a*{xj^), . . . ,a*{xjj} 
for the same indices ji, . . . ,js, because XY{p) and XY (q) have Hnearly inde- 
pendent columns in the same positions. Thus V2{p) and V2{q) hst the images of 
the same set of monomials XjXk under a* and a* respectively. 

Proceeding by induction, if XY{p) and X^ (q) have the same RREF and list 
the images of the same monomials, then XY^^^{p) and Xj^j^(g) also list the images 
of the same monomials. By the assumption Ci+i(p) = Ci+i(g), the matrices 
X]^i{p) and XY^^{q) also have the same RREF. Therefore the kernels of a* 
and a* have the same canonical fc-basis for their degree-i subspaces, completing 
the induction. In particular, the same ideal {fi, . . . ,fs) defines G ■ p and G ■ q. 
Since G is an algebraic group, it follows G ■ p — G ■ q, completing the proof. 

4-2. Complexity Bounds 

The bookkeeping that follows confirms that the complexity of the orbit sep- 
arating algorithm is polynomial in n and N. First, the degree bound N'^M^~"^ 
for a generating set of q requires that we compute products of N^M^~™ degree- 
A^ polynomials fi in k[zi, . . . , z^], for i = 1, . . . , N'^M^~"^ . To this end, compute 
the monomials in the Zj up to degree N ■ iVM^"™, with total complexity 
0(Ai'^('-+i)M^(^~™)). Then multiply /1/2 • • • /» and /,+i to obtain an implicit 
straight-line program for the product of i -I- 1 distinct degree- iV polynomials in 
k[zi, . . . ,Z£], with complex ity 0(2^^~-^i'^^N ^ ^). Fo r details of polynomial multi- 



plication, see Chapter 2 of iBiirgisser et al.l ()1997l ) 



Next consider the sizes of matrices in the algorithm. Recall that for large d, 
H{d) < 0{d™). Hence in iteration «, the matrix Xi has 

h^O {{{i - 1)7V)™ + n ■ [{{i - l)Nr - ((« - 2)N)^]) 

columns from V^, has \B{iN)\ additional columns, and has [iNY rows corre- 
sponding to the monomials in fc[zi, . . . , zt]<iN- Of course, \B{iN)\ — 0{{iNY), 
so the number of rows oiXi is 0{{iNY), and the number of columns is 0{n{iN)"''- 
{iNY) < 0{n{iNY)- Now, computing the tRREF of an s x i matrix has 
complexity 0{st^ + t^). Thus the computation of tRREF(Xi) has complexity 
bounded by 

O {{iNY ■ n'iiNf' + n^iiNf) = O {ri'i^'N^') . 

The above count of the columns of Xi also yields that the computation of the 
kernel of tRREF(Xj) has complexity 0{n'^i'^'^N'^'^) 

In collecting the independent elements of Vi in step (10), the input to the 
procedure S is a fc^ x [iNY matrix, where 

k,^0 (((^ - l)Af)™ + n ■ [{{i - l)iV)"' - {{i - 2)N)"']) < 0{n{iN)'^). 

On an s X i matrix, E has complexity 0{s^t), whence step (10) has complexity 

< 0(n2(iiV)2™ . (iNY). 
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Finally, the polynomial multiplications /i • • • /i proceed through i ^ N^M^ "^ 
with n multiplications for each i. Their total complexity is 

Of the other computations, the programs for the tRREF have the highest 
cost. Summing their complexity from i = 1 to the degree bound, N"^ M^'™ , 
yields the following: 

where, again, N is the maximum polynomial degree of the representation, M 
is a degree bound for a generating set of 1(G) C k[zi, . . . , z^], and under this 
embedding G has dimension m. Since the embedding G ^-> A^ is fixed, we omit 
the constant power of M from the asymptotic complexity. 

Finally, to bound the number of relations that the algorithm computes, we 
sum the column count 0{n{iNY) of the matrices Xi over all iterations i, and 
obtain 

O (nAr'^(''+^'+''Af (^"™'(^+^'') 

polynomials generating the ideal q. In iteration i, such a polynomial has ki < 
0{n{iN)"^) terms, giving a bound for the number of constructible functions that 
the algorithm computes: 

Omitting the powers of M, the main theorem follows. 

5. Conclusion 

Given any representation of a fixed algebraic group, the algorithm writes 
down invariant, constructible functions that separate the orbits of the group 
action. What is more, there are polynomial bounds, in the parameters of the 
representation, for the number of the functions and their total length as straight 
line programs. These bounds describe the complexity of the problem of deter- 
mining if two points lie in the same orbit, by essentially counting the number of 
necessary algebraic operations to perform on the coordinates of the points. Ad- 
ditionally, it emerges that to separate orbits, the "quasi-inverse" is a sufficient 
generalization of the ring of polynomial functions. 
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